According to the WHO, Malaria infects 241 million people a year--leading to over 627 thousand deaths each and every year. In addition to the toll on life, Malaria has wide reaching sociological and economic impact. To quote T. H. Weller, a Nobel laureate in medicine, “It has long been recognized that a malarious community is an impoverished community.” Thus, detecting and treating Malaria stands to improve not only human health but the economic health of communities and entire nations[1].
Malaria is a parasitic disease caused by a single-celled microorganism of genius Plasmodium. The Plasmodium cells are passed into the blood of humans and other animals by mosquito saliva. Once in the blood, the parasites travel into the liver where they mature and reproduce, then infect red blood cells (RBCs). The parasites further multiply in the RBCs, periodically breaking out of their host cells to invade fresh RBCs[2]. This cycle continuously amplifies the number of parasites in an infected person.
Malaria exhibits no symptoms during the liver infection stage--only developing non-specific symptoms once around 100,000 parasites per milliliter of blood[2][3]. Typical diagnostic methods include microscopic examination of blood films, antigen-based rapid diagnostic tests (RDT) and polymerase chain reaction (PCR) to detect Plasmodium DNA. Each method has it’s own limitations and advantages.
Microscopic examination provides information about the extend of the infection however requires slide preparation in a laboratory. RDTs are easily transported and performed but require commercial manufacturing, vary in lot quality and provide less information than microscopy. PCRs can definitively identify a pathogen, however require expense equipment, complexity laboratory procedures, and have poor specificity for active infections. For its balance of availability and diagnostic quality, microscopy remains the gold standard of Malaria diagnosis.
Microscopic examination involves examination of slides, prepared using Giemsa-stained (a nucleic acid stain) films of blood. A “thick film” allows the technician scan many blood cells at a time and a “thin film” to see individually cells and identify the infecting species. Under typical field laboratory conditions, a microscopist can detect parasites when there are at least 100 parasites per milliliter of blood, which is around the lower range of symptomatic infection[4]. With the advent of inexpensive and high quality imaging hardware there exists an opportunity to improve the detection rate of field technicians by utilizing machine vision and deep learning algorithms.
The objective for this project is to build an accurate and efficient machine vision model that can improve the detection rate for a technician. The model is intended to identify whether or not the red blood cell in an image is infected with Malaria. The dataset set consists of 8-bit/color RGB images of varying width and height. The dataset is provided as a compressed zip file and the unzipped directory structure and paths split the dataset between test and train, with subdirectories for parasitize and uninfected images.
Directory Tree for Image Files:
cell_images/
├── test
│ ├── parasitized
│ └── uninfected
└── train
├── parasitized
└── uninfected
There are a total of 24,958 train and 2,600 test images taken from microscopic images ("thin-film" preparation assumed). These images are of the following categories:
A CNN model was chosen based on the nature of the dataset--color images. Consideration was given to how additional training data could be used to build an additive model and how the addition of images might cause an imbalance in the loss function. This was deemed to be out of scope for this project. Transfer learning was utilized to save and compare model architectures which also include the pre-trained VGG16 network.
The model was hosted on the Google Cloud Platform, Colab runtime, which had limitations to the amount of memory available and file access times. Thus, consideration was given to the complexity of the model and file access. Reducing complexety reduced memory allocation and reducing file accesses greatly reduced execution time.
The images were resized and scaled to adhere model input architecture. Care was take to resize images once and save off the results as a single file using the pickle library. Color Space transformations will be evaluated as deem required however there is a motivation to avoid any needless preprocessing that might be learned by the model.
Two methods were explored to remedy training set inbalance--random dropping of inbalanced data and augmentation of positive-negative images thru mirroring and random flip. Each method had staged advantages which could be areas for further research.
We also considered how a model might be transfered to low-compute-capablity edge-computing devices such as smart-phones. To this end, we tuned select models for local hosting by limiting batch size and layer complexity. This became an area of deep interest and a motivation to refine model efficiency and reduce complexity. Although out of scope for this project some care was taken to develop a model and workflow that could be used by the field technician.
We understand that due to the nature of slide preparation that a number of cells will be analysed to make a diagnosis. Thus model accuracy above a certain threshold isn't required for this particalur model. Any such diagnositic threshold is considered out-of-scope for this project.
We shall tune models based on their Test-Accuracy against a validation dataset. We shall evaluate model performance only once and against a full test dataset. Model performance will be evaluated based on precision, recall, and f1-score, as well as, miniminzing the false-negatives (Missclassified infected cells).
Our approach to the problem was mulitstaged:
Stage 1: Exploratory Data Analytics
Stage 2: Data Preprocessing
Stage 3: Model Building
Stage 4:
Stage 5:
General Notes:
"I think we drink virgin blood because it sounds cool."
--Deacon, What We Do in the Shadows
import os
import gc
import re
import sys
import cv2
import pickle
import zipfile
import logging
import numpy as np
import pandas as pd
import seaborn as sns
import sklearn as skl
import tensorflow as tf
import matplotlib.pyplot as plt
from io import BytesIO
from datetime import datetime
from google.colab import drive
from PIL import Image, ImageOps
from tensorflow.python.client import device_lib
# Define Debug Assert
def assertEnd():
assert(False), ('\n' +
'This is the end, beautiful friend\n' +
'This is the end, my only friend\n' +
'The end of our elaborate plans\n'+
'The end of everything that stands\n--The Doors') #
# Mount the Drive
path=[]
LOCAL = False
if LOCAL:
path = './content/'
else:
drive.mount('/content/drive')
path='/content/drive/MyDrive/Colab Notebooks/content/'
Drive already mounted at /content/drive; to attempt to forcibly remount, call drive.mount("/content/drive", force_remount=True).
# Setup Logging to Record results
#
# We're not going to use this at this time, however
# we do want to establish a methodology
# that might be applied in a production environment.
# create logger for this subject matter
logger = logging.getLogger('Malaria_Detection')
logger.setLevel(logging.DEBUG)
# create file handler which logs even debug messages
logfilename = f'logfile{datetime.now().strftime("%Y%m%d_%X")}.txt'
fh = logging.FileHandler(f'{path}/{logfilename}')
fh.setLevel(logging.DEBUG)
# create formatter
formatter = logging.Formatter('%(asctime)s - %(name)s - %(levelname)s - %(message)s')
# add formatter to fh
fh.setFormatter(formatter)
# add handler to logger
logger.addHandler(fh)
# add stream handler
#sh = logging.StreamHandler(sys.stdout)
#logger.addHandler(sh)
logger.debug('='*60)
logger.debug(f'\tLog File:\t{logfilename}')
logger.debug('='*60)
logger.info('This is level: INFO')
logger.warning('This is level: WARNING')
logger.error('This is level: ERROR')
DEBUG:Malaria_Detection:============================================================ DEBUG:Malaria_Detection: Log File: logfile20230411_18:55:01.txt DEBUG:Malaria_Detection:============================================================ INFO:Malaria_Detection:This is level: INFO WARNING:Malaria_Detection:This is level: WARNING ERROR:Malaria_Detection:This is level: ERROR
print('This ought to be seen in the logs too.') # nope this broken
# it's a bit of a loop to print logger messages to stdout
# and stdout into logger (let's not do that!)
#logging.shutdown()
#assertEnd()
This ought to be seen in the logs too.
print(f"Tensor Flow Version: {tf.__version__}")
print(f"Keras Version: {tf.keras.__version__}")
print(f"Python: {sys.version}")
print(f"Pandas: {pd.__version__}")
# Show GPU info or assert
if len(tf.config.list_physical_devices('GPU')) > 0:
g = re.search('physical_device_desc: "(.*)"', str(device_lib.list_local_devices()))
if g:
print(f'GPU Info: {g.groups()[0]}')
else:
assert(False), 'No GPU info found. TPU not supported.'
else:
assert(True), 'GPU is NOT AVAILABLE. ;-(' # disabled for testing
Tensor Flow Version: 2.12.0 Keras Version: 2.12.0 Python: 3.9.16 (main, Dec 7 2022, 01:11:51) [GCC 9.4.0] Pandas: 1.5.3 GPU Info: device: 0, name: Tesla T4, pci bus id: 0000:00:04.0, compute capability: 7.5
The dataset was provided as a zip file. Since Google drive is slow to open files, care is taken to unzip this file only once. Likelywise, each file must be opened during resizing so the resulting numpy arrays are saved as a single pickle file after processing. If the pickle file so present, the arrays are loaded directly from the pickle file, avoiding this lengthy processes.
The extracted folder has different folders for train and test data will contain the different sizes of images for parasitized and uninfected cells within the respective folder name.
The size of all images must be the same and should be converted to 4D arrays so that they can be used as an input for the convolutional neural network. Also, we need to create the labels for both types of images to be able to train and test the model.
Let's do the same for the training data first and then we will use the same code for the test data as well.
SIZE = 72
BRATIO = 0.02 # boarder to image width ratio
EQUAL_ASPECT = False
def process_image(image):
'''
Process image by streching, filling and scaling. Returns processed image.
'''
S = max(image.size)
if not EQUAL_ASPECT: # eg. preserves original aspect... logic made sense at the time. ;-p
image = ImageOps.pad(image, size=(S,S) )
image = ImageOps.expand(image, border=int(S*BRATIO), fill='black')
else:
image = image.resize((S, S))
image = ImageOps.expand(image, border=int(S*BRATIO), fill='black')
image = image.resize((SIZE, SIZE))
#plt.imshow(image)
#plt.show()
return image
# Empty list to store the images and labels (0 - uninfected, 1 - parasitized)
train_images = []
train_labels = []
test_images = []
test_labels = []
# check for pickle file then unpack image into memory, resize, and stack
# delete this file to regenerate
PICKLED = None
if os.path.isfile(f'{path}/celldata.pkl'):
PICKLED = True
else:
PICKLED = False
target_zip = f'{path}/cell_images.zip'
if not PICKLED:
# Injest zip
with zipfile.ZipFile(target_zip) as archive:
for info in archive.infolist():
filename = info.filename
#print(f"Filename: {filename}")
m = re.search(r'cell_images/(test|train)/(parasitized|uninfected).*png$', filename)
if m:
#print(f'{m.group(1)}, {m.group(2)}')
data = archive.read(filename)
b = BytesIO(data)
image = Image.open(b)
image = process_image(image)
#print(len(image))
name = m.group(1)
label = 1 if m.group(2) == 'parasitized' else 0
if name == 'test':
test_images.append(np.array(image))
test_labels.append(label)
else:
train_images.append(np.array(image))
train_labels.append(label)
#assert(False), "FIN."
# This saves lots of time!
# create pickle if not pickled,
# load pickle otherwise
if not PICKLED:
# create pickle file so that we don't do this again
with open(f'{path}/celldata.pkl', 'wb') as db_file:
pickle.dump(obj={'train_images': np.array(train_images),
'train_labels': np.array(train_labels),
'test_images': np.array(test_images),
'test_labels': np.array(test_labels),
'size': SIZE}, file=db_file)
print('Gherkin created.')
PICKLED = True
else:
with open(f'{path}/celldata.pkl', 'rb') as db_file:
db_pkl = pickle.load(db_file)
train_images = db_pkl['train_images']
train_labels = db_pkl['train_labels']
test_images = db_pkl['test_images']
test_labels = db_pkl['test_labels']
SIZE = db_pkl['size']
print('Gherkin injested.')
Gherkin injested.
# Provide a helper function to reload the data
def reload_data():
with open(f'{path}/celldata.pkl', 'rb') as db_file:
db_pkl = pickle.load(db_file)
print('Gherkin injested.')
return (db_pkl['train_images'],
db_pkl['train_labels'],
db_pkl['test_images'],
db_pkl['test_labels'])
train_images, train_labels, test_images, test_labels = reload_data()
Gherkin injested.
len(test_images)
2600
print(f'Train Images: {train_images.shape}\t Test Images: {test_images.shape}')
print(f'Train Labels: {train_labels.shape}\t\t\t Test Labels: {test_labels.shape}')
print(f'Train Images Value Range: {train_images.min()} to {train_images.max()}\t Type:{train_images.dtype}')
Train Images: (24958, 72, 72, 3) Test Images: (2600, 72, 72, 3) Train Labels: (24958,) Test Labels: (2600,) Train Images Value Range: 0 to 255 Type:uint8
# Try to use value_counts to count the values
def count_labels(labels):
'''Returns: infected_count, uninfected_count'''
return labels.sum(), (len(labels) - labels.sum())
print(f'Train: (Infected, Healthy) = {count_labels(train_labels)}')
print(f'Test: (Infected, Healthy) = {count_labels(test_labels)}')
#assertEnd()
Train: (Infected, Healthy) = (12582, 12376) Test: (Infected, Healthy) = (1300, 1300)
def plot_balance(train_labels, test_labels):
plt.subplot(1,2,1)
plt.pie(list(count_labels(train_labels)),
labels = [ 'Infected', 'Healthy'],
autopct='%2.2f%%');
plt.title('Train')
plt.subplot(1,2,2)
plt.pie(list(count_labels(test_labels)),
labels = [ 'Infected', 'Healthy'],
autopct='%2.2f%%');
plt.title('Test');
print(f'Train: (Infected, Healthy) = {count_labels(train_labels)}')
print(f'Test: (Infected, Healthy) = {count_labels(test_labels)}')
plot_balance(train_labels, test_labels)
Train: (Infected, Healthy) = (12582, 12376) Test: (Infected, Healthy) = (1300, 1300)
Images are of 3 colours of varying pixel widths and height, rescaled to 64x64 (or 72x72 when boarder enabled) pixels. Training dataset is comprised of 24958 images and the test set of 2600 images.
Pixels are represented as an 8 bit unsigned integer thus normalizing to a positive float (from 0 to 1) will be required for positive-definates of our model.
The images have been processed by cropping and masking of individual RBCs. Based on our background research[5] these images were likely prepared by a SNN (Segmentation Neural Network).
The number of Healthy and Infected images was roughly equal but rebalancing the training set will be key to loss function performance.
"be polite, be professional, but have a plan to kill everybody you meet"
--James Norman Mattis, Retired 4-Star General, U.S. Marine Corps
Before we rebalance we are going to split off a validation set. We'll use this set to evaluate and tune our model. Using this method we'll maintain the purity of the test set. This propablity a bit overkill, dropping worked equally well.
from sklearn.model_selection import train_test_split
#train_images, train_labels, test_images, test_labels = reload_data() # for testing
split_ratio = 0.2 # 20%
# Unfortunately split does work the way we wanted it to.
# We want the validation to remain a pure balanced sample.
# It doesn't make any sense to augment the validation test,
# thus we shall only augment the training set.
# build a list of balanced random indexes
# We made this function return 2*N entries... it made sense at the time.
def balanced_sample(labels, N):
np.random.seed(42)
index_set_0 = set()
index_set_1 = set()
#N = (6*6)//2
while ( (len(index_set_0) < N) or (len(index_set_1) < N)):
index = int(np.random.randint(0, labels.shape[0], 1))
if train_labels[index] == 0:
if (len(index_set_0) < N):
index_set_0.add(index)
else:
if (len(index_set_1) < N):
index_set_1.add(index)
return list(index_set_0) + list(index_set_1)
# now make a sample dataset
index_list = balanced_sample(train_labels, (len(train_labels)* split_ratio)//2)
validation_images = []
validation_labels = []
for index in index_list:
image = np.array(train_images[index].copy())
validation_images.append(image)
validation_labels.append(train_labels[index])
# spank it on the bottom...
# and that's how new data is borne! :D
validation_images = np.array(validation_images)
validation_labels = np.array(validation_labels)
# now drop the validation indexes from the training set
train_labels = np.delete(train_labels, list(index_list),0)
train_images = np.delete(train_images, list(index_list),0)
plot_balance(train_labels, validation_labels)
plt.show()
print('Note: Test refers to Validation here.')
Train: (Infected, Healthy) = (10087, 9881) Test: (Infected, Healthy) = (2495, 2495)
Note: Test refers to Validation here.
Two method of rebalancing were tested,
We'll drop any test dataset inbalance and augment any train dateset inbalance.
# Method 2-- randomized index with mirrored randomized flip
#train_images, train_labels, test_images, test_labels = reload_data() # for testing
# Consider side-effects
# Is this really better than dropping? We say yes!
# ... As long as the imbalance is small and
# in the direction of infected labels
def balance_agument(train_labels):
# calculate difference
num1, num0 = count_labels(train_labels)
c = num1 - num0
# Recall: 1 <=> infected and 0 <==> healthy
balance = 0
if (c<0):
balance = 1
# build a list of random indexes
index_list = set()
if (abs(c)>0):
print(f'There are {num1-num0} sample inbalance between infected and health images.')
print("Let's fix that with a mirror and random flip...")
while len(index_list) < abs(c):
index = int(np.random.randint(0, train_labels.shape[0], 1))
if train_labels[index] == balance:
index_list.add(index)
# mirror and flip at random our randomly select images
new_images = []
new_labels = []
for index in index_list:
# mirror
image = np.flip(np.array(train_images[index].copy()), axis = 0)
# random flip
if bool(np.random.randint(0,2,1)):
image = np.flip(image, axis = 1)
# spank it on the bottom...
# and that's how new data is borne! :D
new_images.append(image)
new_labels.append(train_labels[index])
return new_images, np.array(new_labels)
new_images, new_labels = balance_agument(train_labels)
train_images = np.vstack((train_images, new_images))
train_labels = np.hstack((train_labels, new_labels))
plot_balance(train_labels, validation_labels)
print('Note: Test refers to Validation here.')
There are 206 sample inbalance between infected and health images. Let's fix that with a mirror and random flip... Train: (Infected, Healthy) = (10087, 10087) Test: (Infected, Healthy) = (2495, 2495) Note: Test refers to Validation here.
# Method 1-- randomized drop
def balance_drop_list(train_labels):
np.random.seed(42)
index_list = set()
# calculate difference
num1, num0 = count_labels(train_labels)
c = num1 - num0
# select whether to drop infected or uninfected image to balance out the data
balance = 1
if (c<0):
balance = 0
if (abs(c)>0):
print(f'There are {num1-num0} sample inbalance between infected and health images.')
print("Let's fix that now...")
while len(index_list) < abs(c):
index = int(np.random.randint(0, train_labels.shape[0], 1))
if train_labels[index] == balance:
index_list.add(index)
return index_list
# We've seen Google drive errors on unzipping hence this extra code.
index_list = balance_drop_list(test_labels)
if len(index_list) > 0:
test_images = np.delete(test_images, list(index_list),0)
test_labels = np.delete(test_labels, list(index_list),0)
plot_balance(train_labels, test_labels)
Train: (Infected, Healthy) = (10087, 10087) Test: (Infected, Healthy) = (1300, 1300)
We choose to split off a balances validation, then augment the remaining training set. The training set had 209 more infected images which we augmented with 209 mirrored and randomly flipped healthy images. We also tested simple dropping but found a slight improvement during fitting with augmentation. It might be aurged that this introduces bias, but we'll leave this as a point of dicussion and debate. We might show experimental proof one way of the other in the final analysis.
Before we rebalance we choose to split off a balanced validation set. We'll use this set to evaluate and tune our model. Using this method we'll maintain the purity of the test set.
Let's visualize the images from the train data, but first let's create a balanced random sample. We do this to reduce memory usage and to minimize data leakage via the mind of the data scientist.
#train_images, train_labels, test_images, test_labels = reload_data() # for testing
# build a list of balanced random indexes
index_list = balanced_sample(train_labels, (6*6)//2)
# now make a sample dataset (6*6)//2
sample_images = []
sample_labels = []
for index in index_list:
image = np.array(train_images[index].copy())
sample_images.append(image)
sample_labels.append(train_labels[index])
# spank it on the bottom...
# and that's how new data is borne! :D
sample_images = np.array(sample_images)
sample_labels = np.array(sample_labels)
print(f'Sample Images: {sample_images.shape}\t Test Images: {sample_images.shape}')
print(f'Sample Labels: {sample_labels.shape}\t\t\t Test Labels: {sample_labels.shape}')
print(f'Sample Images Value Range: {sample_images.min()} to {sample_images.max()}\t Type:{sample_images.dtype}')
Sample Images: (36, 72, 72, 3) Test Images: (36, 72, 72, 3) Sample Labels: (36,) Test Labels: (36,) Sample Images Value Range: 0 to 250 Type:uint8
# this dataset will always be a fixed size
def show_set(images, labels):
N = 6
fig = plt.figure(1, figsize = (12 , 12))
for n in range(1, N**2+1):
plt.subplot(N, N, n)
index = n-1
if labels[index] == 1:
plt.title('parasitized')
else:
plt.title('uninfected')
plt.imshow(images[index])
plt.axis('off')
show_set(sample_images, sample_labels)
Infected cells appear to have a purple spot (Giemsa-stain) assocated with the parasitived plasmid. Heathly cells are mostly of a single color without much gradient. We might consider use of hue normalizion.
As noted previously these images were likely prepared by a SNN (Segmentation Neural Network) which provided cropping and masking of individual RBCs. The images are tightly cropped. We might consider adding a slightly larger black background to avoid our model training on the boarder.
def rgb2hsv(images):
gfx=[] # to hold the HSV image array
for image in images:
#a = cv2.cvtColor(image, cv2.COLOR_BGR2HSV) # why was this wrong in the example?
a = cv2.cvtColor(image, cv2.COLOR_RGB2HSV) # consider how cv2 treats normalized images
gfx.append(a)
return (np.array(gfx)).astype('float32') # hues are angles
def hsv2rgb(images):
gfx=[] # to hold the RGB image array
for image in images:
a = cv2.cvtColor(image, cv2.COLOR_HSV2RGB) # consider how cv2 treats normalized images
gfx.append(a)
return (np.array(gfx)).astype('float32')
def blur(images):
gfx=[] # to hold the Blur image array
for image in images:
a = cv2.GaussianBlur(image, (5, 5), 0)
gfx.append(a)
return np.array(gfx)
def rgb2norm(images):
# Don't normalize twice.
if images.max() > 1:
return (images/255.0).astype('float32')
else:
return images.astype('float32')
# Recall: for labels a 1 is parasitised and 0 is uninfected
#train_images, train_labels, test_images, test_labels = reload_data() # for testing
# Create a lists to store the data
parasitized_data = []
uninfected_data = []
# use positive-definate norm to find mean images
for img, label in zip(rgb2norm(train_images), rgb2norm(train_labels)):
if label == 1:
parasitized_data.append([img])
else:
uninfected_data.append([img])
# find the mean
find_mean_img = lambda full_mat: np.mean(full_mat, axis = 0)[0]
parasitized_mean = find_mean_img(np.array(parasitized_data))
uninfected_mean = find_mean_img(np.array(uninfected_data))
#diff_mean = np.array([1.0,1.0,1.0]) - (uninfected_mean-parasitized_mean)
# Let's call this a dynamic mask (We are making this up.)
diff_mean = (uninfected_mean-parasitized_mean)
diff_mean -= diff_mean.min()
diff_mean *= 1/diff_mean.max()
# plot the result
plt.suptitle('Average and Difference-of-Average')
plt.subplot(1,3,1)
plt.imshow(parasitized_mean)
plt.title('Parasitized')
plt.axis('off')
plt.subplot(1,3,2)
plt.imshow(uninfected_mean)
plt.title('Uninfected')
plt.axis('off')
plt.subplot(1,3,3)
plt.imshow(diff_mean)
plt.title('Difference')
plt.axis('off')
plt.show()
# not sure of utility but it certainly looks cool!
# maybe we could use this as a mask...
diff_mean.min(), diff_mean.max()
(0.0, 1.0)
# HSV
sample_images_hsv = rgb2hsv(rgb2norm(sample_images))/360.0
# viewing hsv would be non-sense but
# we'll do a unity converson back to rgb
show_set(hsv2rgb(sample_images_hsv*360.0), sample_labels)
# Blur
sample_images_blur = blur(rgb2norm(sample_images))
show_set(sample_images_blur, sample_labels)
sample_images_blur.min(), sample_images_blur.max()
(0.0, 0.8737745)
# This could be of some value.
# Those parasites are popping out now.
sample_images_dynamic = rgb2norm(sample_images) * diff_mean
show_set(sample_images_dynamic, sample_labels)
sample_images_dynamic.min(), sample_images_dynamic.max()
(0.0, 0.82986844)
Here we choose to normalize based on the mean color hue. We might have choosen to normalize on the mean image. We'll explore as well if time allows.
Normalization is a method of scaling to shift and scale (min-max scaling).
$$X_{norm}=\frac{X-X_{min}}{X_{max}-X_{min}}$$
Standardization is a method where values are centered around mean with standard deviation. $$z=\frac{x-\mu}{\sigma}$$ Where, $\mu$ is the mean of distribution and $\sigma$ is the standard deviation of the distribution.
A range of 0 to 1 is required for positive definiteness.
# impliment the formulas above
def rgb2std(sample_images, hsv=False):
# Standardization
X = rgb2hsv(sample_images)
mu = np.array([X[:][0].mean(), X[:][1].mean(), X[:][3].mean()])
sigma = np.array([X[:][0].std(), X[:][1].std(), X[:][3].std()])
sample_images_std = np.divide((X-mu),sigma)
# Normalization
X_max = np.array([sample_images_std[:][0].max(), sample_images_std[:][1].max(), sample_images_std[:][3].max()])
X_min = np.array([sample_images_std[:][0].min(), sample_images_std[:][1].min(), sample_images_std[:][3].min()])
X_norm = np.divide(sample_images_std - X_min, X_max - X_min)
sample_images_std = X_norm/X_norm.max() # fix up rounding error
if hsv:
return sample_images_std
else:
return hsv2rgb(sample_images_std)
sample_images_std = rgb2std(sample_images)
show_set(sample_images_std, sample_labels)
sample_images_std.min(), sample_images_std.max()
(0.0051865764, 1.0)
See: Databook[6], Brunton & Kutz
import pywt
A = sample_images[30]
#def image2wavelet(A):
R = np.array([1,0,0])
G = np.array([0,1,0])
B = np.array([0,0,1])
N = []
L = [R,G,B]
C = ['Reds', 'Greens', 'Blues']
#detected_output = cv2.bitwise_and(image, image, mask = mask)
#B = np.mean(A, -1); # Convert RGB to grayscale
fig = plt.figure(1, figsize = (10, 10))
for i, l in enumerate(L):
plt.subplot(1,3,i+1)
D = A * l
B = np.mean(D, -1); # Single Color to grayscale
## Wavelet decomposition (n-level)
n = 4
w = 'db1'
coeffs = pywt.wavedec2(B,wavelet=w,level=n)
# normalize each coefficient array
coeffs[0] /= np.abs(coeffs[0]).max()
for detail_level in range(n):
coeffs[detail_level + 1] = [d/np.abs(d).max() for d in coeffs[detail_level + 1]]
arr, coeff_slices = pywt.coeffs_to_array(coeffs)
#plt.imshow(D)
plt.imshow(arr,cmap=C[i],vmin=-0.25,vmax=0.75)
N.append(arr)
print(arr.shape)
plt.show()
(73, 73) (73, 73) (73, 73)
#np.array([N,N,N]).T.shape
M = (1-np.array(N).T)/2
plt.imshow(M)
M.max()
1.0
def image2wavelet(A):
R = np.array([1,0,0])
G = np.array([0,1,0])
B = np.array([0,0,1])
N = []
L = [R,G,B]
# ugly and slow
for i, l in enumerate(L):
D = A * l
B = np.mean(D, -1); # Single Color to grayscale
## Wavelet decomposition (n-level)
n = 4
w = 'db1'
coeffs = pywt.wavedec2(B,wavelet=w,level=n)
# normalize each coefficient array
coeffs[0] /= np.abs(coeffs[0]).max()
for detail_level in range(n):
coeffs[detail_level + 1] = [d/np.abs(d).max() for d in coeffs[detail_level + 1]]
arr, coeff_slices = pywt.coeffs_to_array(coeffs)
N.append(arr)
return (np.array(N).T + 1)/2
def rgb2wavelet(images):
gfx=[]
for image in images:
a = image2wavelet(image)
gfx.append(a)
return np.array(gfx)
#plt.imshow(image2wavelet(sample_images[30]))
sample_images_wavelet = rgb2wavelet(sample_images)
show_set(sample_images_wavelet, sample_labels)
As we expected, color space transformations are isometric so they have no effect on the model. At best they pull out a layer into the image preprocessing stage--at worst introduce bias. Color space transforms are linear, less machine epsilon error, thus of no predictive value.
We also experimented with hue adjustment but this was of little predictive value thus excluded from further evaluation. Rounding errors caused minor loss of precision.
Blurring is effectively a low-pass filter as in only low spatial frequencies are allowed to the next layer. Assuming a few big features can be used to classify the images this may be effective. Again this is essentially pulling a layer out of the model.
Wavelet transformation performed well with Batch Normalization but poorly with other models.
In the end we choose to simply resize data with no further preprocessing. We want our model to be trained with an unadulterated data so that all hidden features are avaiable for hypertuning. This may aid in additive learning as well.
Two color space transformation were compared for all models. Both transformation return a positive definite array that was required for model convergence.
Simple rescaling
Hue standardized and normalized
We tested various color space transformation. For isometeric transforms we didn't expect any difference and indeed it was small. We did introduce bias using some methods and introduced error due to rounding and machine epsilon.
In short anything we do here could be learned by the model so isn't of much value. We also considered how our model might be further trained in the field. If we were to do standardization based on the train set this could not be learned later.
'''
# rgb2wavelet
train_images = rgb2wavelet(train_images)
validation_images = rgb2wavelet(validation_images)
test_images = rgb2wavelet(test_images)
date_notes = 'Training Dataset: RGB Wavelet'
'''
# positive definiteness
train_images = rgb2norm(train_images)
validation_images = rgb2norm(validation_images)
test_images = rgb2norm(test_images)
date_notes = 'Training Dataset: RGB'
'''
# stardardized and normalized (RGB)
train_images = rgb2std(train_images)
validation_images = rgb2std(validation_images)
test_images = rgb2std(test_images)
date_notes = 'Training Dataset: Standarized RGB'
'''
'''
# stardardized and normalized (HSV)
train_images = rgb2std(train_images, hsv=True)
validation_images = rgb2std(validation_images, hsv=True)
test_images = rgb2std(test_images, hsv=True)
date_notes = 'Training Dataset: Standarized Hue RGB'
'''
print(train_images.max())
1.0
Consider if this is really needed of not. We think not depending on how the model output layer is defined.
from tensorflow.keras.utils import to_categorical
# Encode Labels
train_labels = to_categorical(train_labels, num_classes=2)
validation_labels = to_categorical(validation_labels, num_classes=2)
test_labels = to_categorical(test_labels, num_classes=2)
During Model Tuning we tested out various dataset. This is here to assure the correct set is used for all models. In the end the only transform utilied was image rescaling which was done during the color transformation step above.
See: Transform color space
my_train_images, my_validation_images, my_test_images = (train_images, validation_images, test_images)
# we are low on memory (this will help kill bugs too)
del(train_images)
del(validation_images)
del(test_images)
This Base Model was provided in the low code example and used here as a baseline. We shall use this model for the performance evaluation of our CNN architectures.
from tensorflow.keras import backend
from tensorflow.keras.models import Sequential, load_model
from tensorflow.keras.layers import Input, Conv2D, MaxPooling2D, Dense, Flatten, Dropout
from tensorflow.keras.callbacks import EarlyStopping, ModelCheckpoint
from random import shuffle
from sklearn.metrics import classification_report
from sklearn.metrics import confusion_matrix
The model backend is cleared and garbage collection is perfromed to assure an untrained model is used each time and to reduce memory usage.
def clean_up():
# Fixing the seed
np.random.seed(42)
import random
random.seed(42)
tf.random.set_seed(42)
# Clean start
backend.clear_session()
gc.collect()
clean_up() # clear backend and do garbage collection
model = Sequential()
model.add(Conv2D(filters = 32, kernel_size = 2, padding = "same", activation = "relu", input_shape = (SIZE, SIZE, 3)))
model.add(MaxPooling2D(pool_size = 2))
model.add(Dropout(0.2))
model.add(Conv2D(filters = 32, kernel_size = 2, padding = "same", activation = "relu"))
model.add(MaxPooling2D(pool_size = 2))
model.add(Dropout(0.2))
model.add(Conv2D(filters = 32, kernel_size = 2, padding = "same", activation = "relu"))
model.add(MaxPooling2D(pool_size = 2))
model.add(Dropout(0.2))
model.add(Flatten())
model.add(Dense(512, activation = "relu"))
model.add(Dropout(0.4))
model.add(Dense(2, activation = "softmax")) # 2 represents output layer neurons
model.summary()
Model: "sequential"
_________________________________________________________________
Layer (type) Output Shape Param #
=================================================================
conv2d (Conv2D) (None, 72, 72, 32) 416
max_pooling2d (MaxPooling2D (None, 36, 36, 32) 0
)
dropout (Dropout) (None, 36, 36, 32) 0
conv2d_1 (Conv2D) (None, 36, 36, 32) 4128
max_pooling2d_1 (MaxPooling (None, 18, 18, 32) 0
2D)
dropout_1 (Dropout) (None, 18, 18, 32) 0
conv2d_2 (Conv2D) (None, 18, 18, 32) 4128
max_pooling2d_2 (MaxPooling (None, 9, 9, 32) 0
2D)
dropout_2 (Dropout) (None, 9, 9, 32) 0
flatten (Flatten) (None, 2592) 0
dense (Dense) (None, 512) 1327616
dropout_3 (Dropout) (None, 512) 0
dense_1 (Dense) (None, 2) 1026
=================================================================
Total params: 1,337,314
Trainable params: 1,337,314
Non-trainable params: 0
_________________________________________________________________
model.compile(loss = 'binary_crossentropy', optimizer = 'adam', metrics = ['accuracy'])
callbacks = [EarlyStopping(monitor = 'val_loss', patience = 4),
ModelCheckpoint('.mdl_wts.hdf5', monitor = 'val_loss', save_best_only = True)]
# "Pandemic II Theme Song" by Dark Realm Studios
import IPython
IPython.display.Audio(f'{path}/PandemicII.mp3', autoplay=True)
def load_pickled_model(model_path):
'''
load_pickled_model(), takes model_path as input returns model, history, notes
'''
history = []
model_history_path = model_path + '/history.pkl'
# Note: Delete models directory to rebuild.
if (os.path.exists(model_path) & os.path.isfile(model_history_path)):
model = load_model(model_path)
with open(model_history_path, 'rb') as db_file:
#history = pickle.load(db_file)
db_pkl = pickle.load(db_file)
history = db_pkl['history']
notes = db_pkl['notes']
print('Gherkin injested.')
else:
assert(False), 'Error Loading Model.'
return model, history, notes
def fit_and_pickle(model, model_path, my_train_images, train_labels, callbacks, notes, forcefit=False):
'''
retrives saved model if file present, forcefit=True overrides saved model
returns: model, history
'''
history = []
model_history_path = model_path + '/history.pkl'
# Note: Delete models directory to rebuild.
if (os.path.exists(model_path) & os.path.isfile(model_history_path)) & forcefit!=True:
model = load_model(model_path)
with open(model_history_path, 'rb') as db_file:
#history = pickle.load(db_file)
db_pkl = pickle.load(db_file)
history = db_pkl['history']
notes = db_pkl['notes']
print('Gherkin injested.')
else:
history = model.fit(my_train_images,
train_labels,
batch_size = 32,
callbacks = callbacks,
validation_data=(my_validation_images, validation_labels),
epochs = 20,
verbose = 1)
history = history.history # history, history, history... history
model.save(model_path)
with open(model_history_path, 'wb') as db_file:
#pickle.dump(history, file = db_file)
pickle.dump(obj={'history':history,
'notes':notes}, file=db_file)
print('Gherkin created.')
return model, history
# See how helpful we have become...
help(load_pickled_model)
help(fit_and_pickle)
Help on function load_pickled_model in module __main__:
load_pickled_model(model_path)
load_pickled_model(), takes model_path as input returns model, history, notes
Help on function fit_and_pickle in module __main__:
fit_and_pickle(model, model_path, my_train_images, train_labels, callbacks, notes, forcefit=False)
retrives saved model if file present, forcefit=True overrides saved model
returns: model, history
# Fit the model
notes = f"Model 0: Three Convolution Layers with Dropout, {date_notes}"
model_path = path + 'models/'+ 'model_0'
model, history = fit_and_pickle(model, model_path,my_train_images, train_labels, callbacks, notes)
Gherkin injested.
def plot_accuracy(history):
N = len(history["accuracy"])
plt.figure(figsize = (7, 7))
plt.plot(np.arange(0, N), history["accuracy"], label = "train_accuracy", ls = '--')
plt.plot(np.arange(0, N), history["val_accuracy"], label = "val_accuracy", ls = '--')
plt.title("Accuracy vs Epoch")
plt.xlabel("Epochs")
plt.ylabel("Accuracy")
plt.legend(loc="upper left")
plot_accuracy(history)
Here we use the validation split we need to evaluate the model. We've leave plotting the confustion matrix for the final evaluation so that we don't leak data out our ears.
# be careful of which dataset we are using
accuracy = model.evaluate(my_validation_images, validation_labels, verbose = 1)
print('\n', 'Test_Accuracy:-', round(accuracy[1],2))
156/156 [==============================] - 3s 4ms/step - loss: 0.0798 - accuracy: 0.9729 Test_Accuracy:- 0.97
See: Model Evaluation for a summary of all confusion matrices.
We purposely don't show them here to avoid data leakage.
def plot_confusion_matrix(model, images):
pred = model.predict(images)
pred = np.argmax(pred, axis = 1)
y_true = np.argmax(test_labels, axis = 1)
# Printing the classification report
print(classification_report(y_true, pred, digits=4))
# Plotting the heatmap using confusion matrix
cm = confusion_matrix(y_true, pred)
plt.figure(figsize = (8, 5))
sns.heatmap(cm, annot = True, fmt = '.0f',
xticklabels = ['Uninfected', 'Parasitized'],
yticklabels = ['Uninfected', 'Parasitized'])
plt.ylabel('Actual')
plt.xlabel('Predicted')
plt.show()
# be careful of which dataset we are using...
# it must be normalized in same manner as training set
# Don't peek yet!
#plot_confusion_matrix(model, my_test_images)
We tested this model using the origin resized images along with hue-norm, hsv and blur images. The Origin, Hue-Normalized and Blur images preformed equally as well, with 98% scores across the board for precision, recall, and f1-score. The HSV converted images preformed almost as well with scores of 97% across the board. This is indeed what we might expect for the homomorphic or isomorphic vector spaces. We might consider if the HSV transform has precision loss during type-casting angle conversion due to machine epsilon.
That went suprisingly well! This will be a hard model to beat! Admittedly we borrowed this model from the low-code example. EDA and model tuning aided in model refinement. With balanced dataset the loss funciton performed better (correctly) in all models.
Things to consider: Adding more layers will likely result in overfitting. Thus, we might focus on reducing complexity to impove speed or improve training through augmented image sets. We might consider normalizing image hue based on the averge hue value. Consider non-ismorphic transformations... are these of metric value? Consider image preprocessing as model layer... which layers can we move to preprocessing?
Try improving model performance by adding layers. Consider altering the activation functions. (See: Model 2 for comparision activation functions.)
clean_up() # clear backend and do garbage collection
model = Sequential()
model.add(Input(shape = (SIZE, SIZE, 3))) # let's be a bit verbose
model.add(Conv2D(filters = 16, kernel_size = 2, padding = "same", activation = "relu"))
model.add(MaxPooling2D(pool_size = 2))
model.add(Dropout(0.2))
model.add(Conv2D(filters = 32, kernel_size = 2, padding = "same", activation = "relu"))
model.add(MaxPooling2D(pool_size = 2))
model.add(Dropout(0.1))
model.add(Conv2D(filters = 32, kernel_size = 2, padding = "same", activation = "relu"))
model.add(MaxPooling2D(pool_size = 2))
model.add(Dropout(0.2))
model.add(Conv2D(filters = 32, kernel_size = 2, padding = "same", activation = "relu"))
model.add(MaxPooling2D(pool_size = 2))
model.add(Dropout(0.2))
model.add(Flatten())
model.add(Dense(512, activation = "relu"))
model.add(Dropout(0.4))
model.add(Dense(2, activation = "softmax")) # 2 represents output layer neurons
model.summary()
Model: "sequential"
_________________________________________________________________
Layer (type) Output Shape Param #
=================================================================
conv2d (Conv2D) (None, 72, 72, 16) 208
max_pooling2d (MaxPooling2D (None, 36, 36, 16) 0
)
dropout (Dropout) (None, 36, 36, 16) 0
conv2d_1 (Conv2D) (None, 36, 36, 32) 2080
max_pooling2d_1 (MaxPooling (None, 18, 18, 32) 0
2D)
dropout_1 (Dropout) (None, 18, 18, 32) 0
conv2d_2 (Conv2D) (None, 18, 18, 32) 4128
max_pooling2d_2 (MaxPooling (None, 9, 9, 32) 0
2D)
dropout_2 (Dropout) (None, 9, 9, 32) 0
conv2d_3 (Conv2D) (None, 9, 9, 32) 4128
max_pooling2d_3 (MaxPooling (None, 4, 4, 32) 0
2D)
dropout_3 (Dropout) (None, 4, 4, 32) 0
flatten (Flatten) (None, 512) 0
dense (Dense) (None, 512) 262656
dropout_4 (Dropout) (None, 512) 0
dense_1 (Dense) (None, 2) 1026
=================================================================
Total params: 274,226
Trainable params: 274,226
Non-trainable params: 0
_________________________________________________________________
model.compile(loss = 'binary_crossentropy', optimizer = 'adam', metrics = ['accuracy'])
callbacks = [EarlyStopping(monitor = 'val_loss', patience = 4),
ModelCheckpoint('.mdl_wts.hdf5', monitor = 'val_loss', save_best_only = True)]
notes = f"Model 1: Four Convolution Layers with Dropout, {date_notes}"
model_path = path + 'models/'+ 'model_1'
model, history = fit_and_pickle(model, model_path,my_train_images, train_labels, callbacks, notes)
Gherkin injested.
plot_accuracy(history)
accuracy = model.evaluate(my_validation_images, validation_labels, verbose = 1)
print('\n', f'Test_Accuracy:- {round(accuracy[1],2)}')
156/156 [==============================] - 1s 4ms/step - loss: 0.0948 - accuracy: 0.9691 Test_Accuracy:- 0.97
See: Model Evaluation for a summary of all confusion matrices.
We purposely don't show them here to avoid data leakage.
# No peeking!
#plot_confusion_matrix(model, my_test_images)
This model appears to perform slightly better with the origin data than Model 0. This model quickly converged to the training accuracy. This might indicate overfitting.
Now let's build a model with LeakyRelu as the activation function
Let us try to build a model using BatchNormalization and using LeakyRelu as our activation function.
from tensorflow.keras.layers import BatchNormalization, LeakyReLU, ReLU
clean_up() # clear backend and do garbage collection
use_relu = False
model = Sequential()
model.add(Input(shape = (SIZE, SIZE, 3))) # let's be a bit verbose
model.add(BatchNormalization()) # reparametrizing... in case we didn't already (we did)
model.add(Conv2D(filters = 16, kernel_size = 2, padding = "same"))
model.add(ReLU()) if use_relu else model.add(LeakyReLU(alpha=0.2))
model.add(MaxPooling2D(pool_size = 2))
model.add(Dropout(0.2))
model.add(Conv2D(filters = 32, kernel_size = 2, padding = "same"))
model.add(ReLU()) if use_relu else model.add(LeakyReLU(alpha=0.2))
model.add(MaxPooling2D(pool_size = 2))
model.add(Dropout(0.2))
model.add(Conv2D(filters = 32, kernel_size = 2, padding = "same"))
model.add(ReLU()) if use_relu else model.add(LeakyReLU(alpha=0.2))
model.add(MaxPooling2D(pool_size = 2))
model.add(Dropout(0.2))
model.add(Flatten())
model.add(Dense(512))
model.add(ReLU()) if use_relu else model.add(LeakyReLU(alpha=0.1))
model.add(Dropout(0.4))
model.add(Dense(2, activation = "softmax")) # 2 represents output layer neurons
model.summary()
Model: "sequential"
_________________________________________________________________
Layer (type) Output Shape Param #
=================================================================
batch_normalization (BatchN (None, 72, 72, 3) 12
ormalization)
conv2d (Conv2D) (None, 72, 72, 16) 208
leaky_re_lu (LeakyReLU) (None, 72, 72, 16) 0
max_pooling2d (MaxPooling2D (None, 36, 36, 16) 0
)
dropout (Dropout) (None, 36, 36, 16) 0
conv2d_1 (Conv2D) (None, 36, 36, 32) 2080
leaky_re_lu_1 (LeakyReLU) (None, 36, 36, 32) 0
max_pooling2d_1 (MaxPooling (None, 18, 18, 32) 0
2D)
dropout_1 (Dropout) (None, 18, 18, 32) 0
conv2d_2 (Conv2D) (None, 18, 18, 32) 4128
leaky_re_lu_2 (LeakyReLU) (None, 18, 18, 32) 0
max_pooling2d_2 (MaxPooling (None, 9, 9, 32) 0
2D)
dropout_2 (Dropout) (None, 9, 9, 32) 0
flatten (Flatten) (None, 2592) 0
dense (Dense) (None, 512) 1327616
leaky_re_lu_3 (LeakyReLU) (None, 512) 0
dropout_3 (Dropout) (None, 512) 0
dense_1 (Dense) (None, 2) 1026
=================================================================
Total params: 1,335,070
Trainable params: 1,335,064
Non-trainable params: 6
_________________________________________________________________
model.compile(loss = 'binary_crossentropy', optimizer = 'adam', metrics = ['accuracy'])
callbacks = [EarlyStopping(monitor = 'val_loss', patience = 4),
ModelCheckpoint('.mdl_wts.hdf5', monitor = 'val_loss', save_best_only = True)]
notes = f"Model 2: Three Convolution Layers with Dropout and first-layer batch normilization, {date_notes}"
model_path = path + 'models/'+ 'model_2'
model, history = fit_and_pickle(model, model_path,my_train_images, train_labels, callbacks, notes)
Gherkin injested.
plot_accuracy(history)
accuracy = model.evaluate(my_validation_images, validation_labels, verbose = 1)
print('\n', f'Test_Accuracy:- {round(accuracy[1],2)}')
156/156 [==============================] - 1s 5ms/step - loss: 0.0985 - accuracy: 0.9713 Test_Accuracy:- 0.97
This model with ReLU performed slightly better than the base model trained on the RGB dataset. LeakyReLU performed slightly less well.
See: Model Evaluation for a summary of all confusion matrices.
We purposely don't show them here to avoid data leakage.
# Don't peek yet!
#plot_confusion_matrix(model, my_test_images)
Can we improve the model with Image Data Augmentation?
References for image data augmentation:
Here we shall use the training dataset to augment. We should consider using the original data files here, however in the interest of breivity and liberty simply use a 3rd of training set to create augmented value.
# Let's play with our sample to find a good balance
from tensorflow.keras.preprocessing.image import ImageDataGenerator
# Using ImageDataGenerator to generate images
train_datagen = ImageDataGenerator(
horizontal_flip = True, # base
zoom_range = 0.03, # base
rotation_range = 30) # base
# Flowing training images using train_datagen generator
train_generator = train_datagen.flow(x = sample_images, y = sample_labels, batch_size = 64, seed = 42, shuffle = True)
# Use the Generator and Show our sample
aug_images, aug_labels = next(train_generator)
show_set(rgb2norm(aug_images), rgb2norm(aug_labels) )
Note: This augmentation is invalid for our 2D wavlet dataset. For that dataset we must augment before transforming to the frequency domain. Colab will run out of memory with anymore than 1/2 the train dataset so that's out max value for now.
#my_train_images, my_validation_images, my_test_images
print(my_train_images.shape, train_labels.shape)
# Prepare the augmented data
batch_size = int(train_labels.shape[0]*1/2) # we want mostly pure images... or do we?
gen1 = train_datagen.flow(x = my_train_images, y = train_labels, batch_size = batch_size, seed = 42, shuffle = True)
new_images, new_labels = next(gen1)
#new_labels = to_categorical(new_labels, num_classes=2)
# stack'em
#train_images = np.vstack((train_images, new_images))
#train_labels = np.vstack((train_labels, new_labels))
print(new_images.shape, new_labels.shape)
(20174, 72, 72, 3) (20174, 2) (10087, 72, 72, 3) (10087, 2)
#check balance and drop
def drop2balance(new_images, new_labels):
np.random.seed(42)
# calculate difference
c = new_labels[:,0].sum() - new_labels[:,1].sum()
# select whether to drop infected or uninfected image to balance out the data
balance = 1
if (c<0):
balance = 0
if (abs(c)>0):
print(f'There are {c} sample inbalance between infected and health images.')
print("Let's fix that now...")
index_list = set()
while len(index_list) < abs(c):
index = int(np.random.randint(0, new_labels.shape[0], 1))
if new_labels[index,0] == balance:
index_list.add(index)
return (np.delete(new_images, list(index_list),0), np.delete(new_labels, list(index_list),0))
new_images, new_labels = drop2balance(new_images, new_labels)
print(new_labels[:,0].sum()-new_labels[:,1].sum())
print(new_images.shape, new_labels.shape)
There are -47.0 sample inbalance between infected and health images. Let's fix that now... 0.0 (10040, 72, 72, 3) (10040, 2)
We found that ImageDataGenerator() doesn't provide a balanced data set. We dropped images to balance the generated dataset. Consider using featurewise_std_normalization and samplewise_std_normalization, if time allows.
# we do this for housekeeping purposes, but it nolonger needed
#my_train_images, my_validation_images, my_test_images = (
# train_images, validation_images, test_images)
We'll reuse the Model 2 but since we are training it differently we shall reproduce the arch again.
clean_up() # clear backend and do garbage collection
model = Sequential()
model.add(Input(shape = (SIZE, SIZE, 3))) # let's be a bit verbose
model.add(BatchNormalization()) # reparametrizing... in case we didn't already (we did)
model.add(Conv2D(filters = 16, kernel_size = 2, padding = "same", activation = "relu"))
model.add(MaxPooling2D(pool_size = 2))
model.add(Dropout(0.2))
model.add(Conv2D(filters = 32, kernel_size = 2, padding = "same", activation = "relu"))
model.add(MaxPooling2D(pool_size = 2))
model.add(Dropout(0.2))
model.add(Conv2D(filters = 32, kernel_size = 2, padding = "same", activation = "relu"))
model.add(MaxPooling2D(pool_size = 2))
model.add(Dropout(0.2))
model.add(Flatten())
model.add(Dense(512, activation = "relu"))
model.add(Dropout(0.4))
model.add(Dense(2, activation = "softmax")) # 2 represents output layer neurons
model.summary()
Model: "sequential"
_________________________________________________________________
Layer (type) Output Shape Param #
=================================================================
batch_normalization (BatchN (None, 72, 72, 3) 12
ormalization)
conv2d (Conv2D) (None, 72, 72, 16) 208
max_pooling2d (MaxPooling2D (None, 36, 36, 16) 0
)
dropout (Dropout) (None, 36, 36, 16) 0
conv2d_1 (Conv2D) (None, 36, 36, 32) 2080
max_pooling2d_1 (MaxPooling (None, 18, 18, 32) 0
2D)
dropout_1 (Dropout) (None, 18, 18, 32) 0
conv2d_2 (Conv2D) (None, 18, 18, 32) 4128
max_pooling2d_2 (MaxPooling (None, 9, 9, 32) 0
2D)
dropout_2 (Dropout) (None, 9, 9, 32) 0
flatten (Flatten) (None, 2592) 0
dense (Dense) (None, 512) 1327616
dropout_3 (Dropout) (None, 512) 0
dense_1 (Dense) (None, 2) 1026
=================================================================
Total params: 1,335,070
Trainable params: 1,335,064
Non-trainable params: 6
_________________________________________________________________
model.compile(loss = 'binary_crossentropy', optimizer = 'adam', metrics = ['accuracy'])
callbacks = [EarlyStopping(monitor = 'val_loss', patience = 4),
ModelCheckpoint('.mdl_wts.hdf5', monitor = 'val_loss', save_best_only = True)]
notes = f"Model 3: Three Convolution Layers with Dropout and first-layer batch normilization, {date_notes} (Augmented)"
model_path = path + 'models/'+ 'model_3'
model, history = fit_and_pickle(model, model_path,
np.vstack((my_train_images, new_images)),
np.vstack((train_labels, new_labels)),
callbacks, notes)
Gherkin injested.
plot_accuracy(history)
accuracy = model.evaluate(my_validation_images, validation_labels, verbose = 1)
print('\n', f'Test_Accuracy:- {round(accuracy[1],2)}')
156/156 [==============================] - 1s 4ms/step - loss: 0.0664 - accuracy: 0.9760 Test_Accuracy:- 0.98
See: Model Evaluation for a summary of all confusion matrices.
We purposely don't show them here to avoid data leakage.
# Don't peek yet!
#plot_confusion_matrix(model, my_test_images)
We found that ImageDataGenerator() doesn't provide a balanced data set. We dropped images to balance the generated dataset. We also found that too many augmented images resulted in overfitting.
Now, let us try to use a pretrained model like VGG16 and check how it performs on our data.
from tensorflow.keras.applications.vgg16 import VGG16
from tensorflow.keras import Model
clean_up() # clear backend and do garbage collection
vgg = VGG16(include_top = False, weights = 'imagenet', input_shape = (SIZE, SIZE, 3))
vgg.summary()
# Select our tranfer layer
transfer_layer = vgg.get_layer('block3_pool')
vgg.trainable = False
# Build our classifier up (we'll use same as in the previous models)
x = Flatten()(transfer_layer.output) # Flatten the output from the 3rd block of the VGG16 model
x = BatchNormalization()(x)
x = Dense(512, activation = 'relu')(x)
x = Dropout(0.2)(x)
x = Flatten()(x)
x = Dense(512, activation = 'relu')(x)
x = Dropout(0.4)(x)
pred = Dense(2, activation = 'softmax')(x)
model = Model(vgg.input, pred) # Initializing the model
model.summary()
Model: "vgg16"
_________________________________________________________________
Layer (type) Output Shape Param #
=================================================================
input_1 (InputLayer) [(None, 72, 72, 3)] 0
block1_conv1 (Conv2D) (None, 72, 72, 64) 1792
block1_conv2 (Conv2D) (None, 72, 72, 64) 36928
block1_pool (MaxPooling2D) (None, 36, 36, 64) 0
block2_conv1 (Conv2D) (None, 36, 36, 128) 73856
block2_conv2 (Conv2D) (None, 36, 36, 128) 147584
block2_pool (MaxPooling2D) (None, 18, 18, 128) 0
block3_conv1 (Conv2D) (None, 18, 18, 256) 295168
block3_conv2 (Conv2D) (None, 18, 18, 256) 590080
block3_conv3 (Conv2D) (None, 18, 18, 256) 590080
block3_pool (MaxPooling2D) (None, 9, 9, 256) 0
block4_conv1 (Conv2D) (None, 9, 9, 512) 1180160
block4_conv2 (Conv2D) (None, 9, 9, 512) 2359808
block4_conv3 (Conv2D) (None, 9, 9, 512) 2359808
block4_pool (MaxPooling2D) (None, 4, 4, 512) 0
block5_conv1 (Conv2D) (None, 4, 4, 512) 2359808
block5_conv2 (Conv2D) (None, 4, 4, 512) 2359808
block5_conv3 (Conv2D) (None, 4, 4, 512) 2359808
block5_pool (MaxPooling2D) (None, 2, 2, 512) 0
=================================================================
Total params: 14,714,688
Trainable params: 14,714,688
Non-trainable params: 0
_________________________________________________________________
Model: "model"
_________________________________________________________________
Layer (type) Output Shape Param #
=================================================================
input_1 (InputLayer) [(None, 72, 72, 3)] 0
block1_conv1 (Conv2D) (None, 72, 72, 64) 1792
block1_conv2 (Conv2D) (None, 72, 72, 64) 36928
block1_pool (MaxPooling2D) (None, 36, 36, 64) 0
block2_conv1 (Conv2D) (None, 36, 36, 128) 73856
block2_conv2 (Conv2D) (None, 36, 36, 128) 147584
block2_pool (MaxPooling2D) (None, 18, 18, 128) 0
block3_conv1 (Conv2D) (None, 18, 18, 256) 295168
block3_conv2 (Conv2D) (None, 18, 18, 256) 590080
block3_conv3 (Conv2D) (None, 18, 18, 256) 590080
block3_pool (MaxPooling2D) (None, 9, 9, 256) 0
flatten (Flatten) (None, 20736) 0
batch_normalization (BatchN (None, 20736) 82944
ormalization)
dense (Dense) (None, 512) 10617344
dropout (Dropout) (None, 512) 0
flatten_1 (Flatten) (None, 512) 0
dense_1 (Dense) (None, 512) 262656
dropout_1 (Dropout) (None, 512) 0
dense_2 (Dense) (None, 2) 1026
=================================================================
Total params: 12,699,458
Trainable params: 10,922,498
Non-trainable params: 1,776,960
_________________________________________________________________
model.compile(loss = 'binary_crossentropy', optimizer = 'adam', metrics = ['accuracy'])
callbacks = [EarlyStopping(monitor = 'val_loss', patience = 4),
ModelCheckpoint('.mdl_wts.hdf5', monitor = 'val_loss', save_best_only = True)]
notes = f"Model 4: VGG16 Transfered at block3_pool Layer with two dense layers, {date_notes}"
model_path = path + 'models/'+ 'model_4'
model, history = fit_and_pickle(model, model_path,my_train_images, train_labels, callbacks, notes)
Gherkin injested.
plot_accuracy(history)
accuracy = model.evaluate(my_validation_images, validation_labels, verbose = 1)
print('\n', f'Test_Accuracy:- {round(accuracy[1],2)}')
156/156 [==============================] - 3s 16ms/step - loss: 0.1721 - accuracy: 0.9547 Test_Accuracy:- 0.95
See: Model Evaluation for a summary of all confusion matrices.
We purposely don't show them here to avoid data leakage.
# Don't peek yet!
#plot_confusion_matrix(model, my_test_images)
This model trained in fewer epoches that the other models. The the validation and train curves indicate that model is well fitted. We might try to tune the loss metric if time allows.
We migth consider adding a bottleneck in the middle of this model to create a latent space thur which the signal must pass. Let's rework this model for as a Autoencoder to show what we've learned up until now.
clean_up() # clear backend and do garbage collection
model = Sequential()
model.add(Input(shape = (SIZE, SIZE, 3))) # let's be a bit verbose
model.add(BatchNormalization()) # reparametrizing... in case we didn't already (we did)
model.add(Conv2D(filters = 16, kernel_size = 2, padding = "same"))
#model.add(Dense(256)) # Lol, this is huge! We are building a bug eye though.
model.add(Dense(64))
model.add(ReLU())
model.add(MaxPooling2D(pool_size = 2))
model.add(Dropout(0.2))
model.add(Dense(128))
model.add(Conv2D(filters = 16, kernel_size = 2, padding = "same"))
model.add(ReLU())
model.add(MaxPooling2D(pool_size = 8))
model.add(Dropout(0.2))
model.add(Conv2D(filters = 4, kernel_size = 2, padding = "same"))
model.add(Dense(25))
model.add(ReLU())
model.add(MaxPooling2D(pool_size = 2))
model.add(Dropout(0.2))
model.add(Conv2D(filters = 16, kernel_size = 2, padding = "same"))
model.add(Dense(64))
model.add(ReLU())
model.add(Flatten())
model.add(ReLU())
model.add(Dropout(0.1))
model.add(Dense(2, activation = "softmax")) # 2 represents output layer neurons
model.summary()
Model: "sequential"
_________________________________________________________________
Layer (type) Output Shape Param #
=================================================================
batch_normalization (BatchN (None, 72, 72, 3) 12
ormalization)
conv2d (Conv2D) (None, 72, 72, 16) 208
dense (Dense) (None, 72, 72, 64) 1088
re_lu (ReLU) (None, 72, 72, 64) 0
max_pooling2d (MaxPooling2D (None, 36, 36, 64) 0
)
dropout (Dropout) (None, 36, 36, 64) 0
dense_1 (Dense) (None, 36, 36, 128) 8320
conv2d_1 (Conv2D) (None, 36, 36, 16) 8208
re_lu_1 (ReLU) (None, 36, 36, 16) 0
max_pooling2d_1 (MaxPooling (None, 4, 4, 16) 0
2D)
dropout_1 (Dropout) (None, 4, 4, 16) 0
conv2d_2 (Conv2D) (None, 4, 4, 4) 260
dense_2 (Dense) (None, 4, 4, 25) 125
re_lu_2 (ReLU) (None, 4, 4, 25) 0
max_pooling2d_2 (MaxPooling (None, 2, 2, 25) 0
2D)
dropout_2 (Dropout) (None, 2, 2, 25) 0
conv2d_3 (Conv2D) (None, 2, 2, 16) 1616
dense_3 (Dense) (None, 2, 2, 64) 1088
re_lu_3 (ReLU) (None, 2, 2, 64) 0
flatten (Flatten) (None, 256) 0
re_lu_4 (ReLU) (None, 256) 0
dropout_3 (Dropout) (None, 256) 0
dense_4 (Dense) (None, 2) 514
=================================================================
Total params: 21,439
Trainable params: 21,433
Non-trainable params: 6
_________________________________________________________________
model.compile(loss = 'binary_crossentropy', optimizer = 'adam', metrics = ['accuracy'])
callbacks = [EarlyStopping(monitor = 'val_loss', patience = 4),
ModelCheckpoint('.mdl_wts.hdf5', monitor = 'val_loss', save_best_only = True)]
notes = f"Model 5: Autoencoder!!, {date_notes}"
model_path = path + 'models/'+ 'model_5'
model, history = fit_and_pickle(model, model_path, my_train_images, train_labels, callbacks, notes, forcefit=False)
Gherkin injested.
plot_accuracy(history)
accuracy = model.evaluate(my_validation_images, validation_labels, verbose = 1)
print('\n', f'Test_Accuracy:- {round(accuracy[1],2)}')
156/156 [==============================] - 2s 12ms/step - loss: 0.0684 - accuracy: 0.9747 Test_Accuracy:- 0.97
See: Model Evaluation for a summary of all confusion matrices.
We purposely don't show them here to avoid data leakage.
# no Peeking allowed
#plot_confusion_matrix(model, my_test_images)
This model has an order of magitude fewer parameters that the previous models. While this model takes longer to train it preforms superior to the next best model, Model 1. Fewer parameters mean this model might be better suited for edge computing devices. This model also improved with augmented data. This might indicate that the model has the capacity to learn from additive training. This model performed the best in the area of minimizing false-negatives, beating the next closest model by more that 2.5 times.
See also: Model comparison (below)
In this section we compare all models side-by-side. All models were trained and evaluated on RGB rescaled datasets. Hue standardized and 2D Wavelet RGB datasets were also tested for all models. The performance scores for the alternate datasets were lower for all models, thus excluced for brevity and clarity.
!pip install keras_flops > /dev/null
from keras_flops import get_flops
# Retrive our models, histories and notes.
model_paths = [path + 'models/'+ f'model_{i}' for i in range(6)]
models = []
histories = []
notes = []
summary = []
for p in model_paths:
model, history, note = load_pickled_model(p)
models.append(model)
histories.append(history)
notes.append(note)
flops = get_flops(model, batch_size=1)
#model.summary()
S = []
model.summary(print_fn=lambda x: S.append(x))
S = "\n".join(S)
#Total params:
param_re = re.search(r'Total params: (.*)', S)
param = 0
if param_re:
param = int(param_re.group(1).replace(',',''))
print(f'{note}: \n\tTotal Parameters: {param/10**6:.03} M-params,\t Flops: {flops/10**9:.03} G-FLOPs')
summary.append((param/10**6, flops/10**9))
WARNING:tensorflow:From /usr/local/lib/python3.9/dist-packages/tensorflow/python/ops/nn_ops.py:5250: tensor_shape_from_node_def_name (from tensorflow.python.framework.graph_util_impl) is deprecated and will be removed in a future version. Instructions for updating: This API was designed for TensorFlow v1. See https://www.tensorflow.org/guide/migrate for instructions on how to migrate your code to TensorFlow v2.
Gherkin injested. Model 0: Three Convolution Layers with Dropout, Training Dataset: RGB: Total Parameters: 1.34 M-params, Flops: 0.0203 G-FLOPs Gherkin injested. Model 1: Four Convolution Layers with Dropout, Training Dataset: RGB: Total Parameters: 0.274 M-params, Flops: 0.0114 G-FLOPs Gherkin injested. Model 2: Three Convolution Layers with Dropout and first-layer batch normilization, Training Dataset: RGB: Total Parameters: 1.34 M-params, Flops: 0.0129 G-FLOPs Gherkin injested. Model 3: Three Convolution Layers with Dropout and first-layer batch normilization, Training Dataset: RGB (Augmented): Total Parameters: 1.34 M-params, Flops: 0.0129 G-FLOPs Gherkin injested. Model 4: VGG16 Transfered at block3_pool Layer with two dense layers, Training Dataset: RGB: Total Parameters: 12.7 M-params, Flops: 1.95 G-FLOPs Gherkin injested. Model 5: Autoencoder!!, Training Dataset: RGB: Total Parameters: 0.0214 M-params, Flops: 0.0561 G-FLOPs
#plt.scatter(summary)
fig, ax = plt.subplots()
for i, O in enumerate(summary):
#print(O)
if i==3:
plt.scatter(O[0], O[1], marker = '*', s= 100, alpha=0.5)
ax.annotate(f'Model {i}', O, xycoords='data',
xytext=(0, -9), textcoords='offset points')
else:
plt.scatter(O[0], O[1], s=float(O[0]*O[1])*100+10, alpha=0.75)
ax.annotate(f'Model {i}', O, xycoords='data',
xytext=(0, 3), textcoords='offset points')
plt.xscale('log')
plt.yscale('log')
plt.title('Total Parameters vs Flops')
plt.xlabel('Parameters (Mega)')
plt.ylabel('Flops (Giga)')
plt.grid()
Observations Note that Model 2 and 3 are identical. The model with the most parameters and highest number of floating pointer operations is Model 4 (VGG16). Model 1 has the lowest number of floating-point operations and Model 5 has the lowest number of parameters.
We might consider how our GPU does floating-point operation and if there is a optimal datatype (float16?) for this model. Typecasting has been noted to be 'expensive' in our former research and testing.
Below we see that some model trained more quickly (few epochs) than others, which the base-model taking the most epochs to train and VGG-model the least epochs. The base-model with batch normilization trained almost as quickly as the VGG-model.
plt.figure(figsize = (14, 7))
for i,h in enumerate(histories):
history = h
plt.suptitle("Accuracy vs Epoch")
plt.subplot(1,len(histories),i+1)
N = len(history["accuracy"])
plt.ylim((0.92,.99))
plt.xlim((0,14))
plt.plot(np.arange(0, N), history["accuracy"], label = "train_accuracy", ls = '--')
plt.plot(np.arange(0, N), history["val_accuracy"], label = "val_accuracy", ls = '--')
plt.title(f'Model {i}')
plt.grid(True)
if i==0:
plt.xlabel("Epochs")
plt.ylabel("Accuracy")
else:
plt.yticks([])
plt.legend(loc="lower right");
plt.show()
print('Recall our Models:')
for n in notes:
print(f'\t{n}')
Recall our Models: Model 0: Three Convolution Layers with Dropout, Training Dataset: RGB Model 1: Four Convolution Layers with Dropout, Training Dataset: RGB Model 2: Three Convolution Layers with Dropout and first-layer batch normilization, Training Dataset: RGB Model 3: Three Convolution Layers with Dropout and first-layer batch normilization, Training Dataset: RGB (Augmented) Model 4: VGG16 Transfered at block3_pool Layer with two dense layers, Training Dataset: RGB Model 5: Autoencoder!!, Training Dataset: RGB
def plot_confusion_matrix(model, images):
pred = model.predict(images)
pred = np.argmax(pred, axis = 1)
y_true = np.argmax(test_labels, axis = 1)
# Printing the classification report
print(classification_report(y_true, pred, digits=4))
# Plotting the heatmap using confusion matrix
cm = confusion_matrix(y_true, pred)
plt.figure(figsize = (8, 5))
sns.heatmap(cm, annot = True, fmt = '.0f',
xticklabels = ['Uninfected', 'Parasitized'],
yticklabels = ['Uninfected', 'Parasitized'])
plt.ylabel('Actual')
plt.xlabel('Predicted')
plt.show()
Our model are evaluated on precision, recall, and f1-score, as well as, miniminzing the false-negatives (Missclassified infected cells).
# Use our models to make our predictions
Y = np.argmax(test_labels, axis = 1)
Y_hat = []
for model in models:
y = model.predict(my_test_images)
Y_hat.append(np.argmax(y, axis = 1))
82/82 [==============================] - 0s 4ms/step 82/82 [==============================] - 1s 3ms/step 82/82 [==============================] - 0s 4ms/step 82/82 [==============================] - 0s 3ms/step 82/82 [==============================] - 1s 14ms/step 82/82 [==============================] - 1s 13ms/step
# Evaluate our models based on precision, recall, and f1-score
f1 = []
for i,model in enumerate(models):
print()
print('='*50)
print(notes[i])
print('='*50)
S = classification_report(Y, Y_hat[i], digits=4)
g = re.search(r'weighted avg\s*(0.\d*)\s*(0.\d*)\s*(0.\d*)', S)
if g:
#print(g.group(1), g.group(2),g.group(3))
f1.append(float(g.group(3)))
print(S)
==================================================
Model 0: Three Convolution Layers with Dropout, Training Dataset: RGB
==================================================
precision recall f1-score support
0 0.9720 0.9885 0.9802 1300
1 0.9883 0.9715 0.9798 1300
accuracy 0.9800 2600
macro avg 0.9801 0.9800 0.9800 2600
weighted avg 0.9801 0.9800 0.9800 2600
==================================================
Model 1: Four Convolution Layers with Dropout, Training Dataset: RGB
==================================================
precision recall f1-score support
0 0.9714 0.9923 0.9817 1300
1 0.9921 0.9708 0.9813 1300
accuracy 0.9815 2600
macro avg 0.9818 0.9815 0.9815 2600
weighted avg 0.9818 0.9815 0.9815 2600
==================================================
Model 2: Three Convolution Layers with Dropout and first-layer batch normilization, Training Dataset: RGB
==================================================
precision recall f1-score support
0 0.9698 0.9892 0.9794 1300
1 0.9890 0.9692 0.9790 1300
accuracy 0.9792 2600
macro avg 0.9794 0.9792 0.9792 2600
weighted avg 0.9794 0.9792 0.9792 2600
==================================================
Model 3: Three Convolution Layers with Dropout and first-layer batch normilization, Training Dataset: RGB (Augmented)
==================================================
precision recall f1-score support
0 0.9861 0.9846 0.9854 1300
1 0.9846 0.9862 0.9854 1300
accuracy 0.9854 2600
macro avg 0.9854 0.9854 0.9854 2600
weighted avg 0.9854 0.9854 0.9854 2600
==================================================
Model 4: VGG16 Transfered at block3_pool Layer with two dense layers, Training Dataset: RGB
==================================================
precision recall f1-score support
0 0.9345 0.9762 0.9549 1300
1 0.9750 0.9315 0.9528 1300
accuracy 0.9538 2600
macro avg 0.9548 0.9538 0.9538 2600
weighted avg 0.9548 0.9538 0.9538 2600
==================================================
Model 5: Autoencoder!!, Training Dataset: RGB
==================================================
precision recall f1-score support
0 0.9808 0.9846 0.9827 1300
1 0.9846 0.9808 0.9827 1300
accuracy 0.9827 2600
macro avg 0.9827 0.9827 0.9827 2600
weighted avg 0.9827 0.9827 0.9827 2600
# Compare f1-scores
labels = [f'Model {i}' for i in range(len(f1))]
plt.bar(labels,f1)
plt.ylim(0.95,.99)
plt.title('F1-Score Comparison')
plt.ylabel('F1-Score')
plt.xlabel('Model')
plt.grid()
# Visually compare our models based using a confusion matrix
plt.figure(figsize = (20, 5))
plt.suptitle("Confusion Matrix Comparison")
for i in range(len(models)):
cm = confusion_matrix(Y, Y_hat[i])
ax = plt.subplot(1,len(models)+1,i+1)
ax.set_box_aspect(1)
sns.heatmap(cm, annot = True, fmt = '.0f', cbar=False, cmap='Blues',
xticklabels = ['Uninfected', 'Parasitized'],
yticklabels = ['Uninfected', 'Parasitized'])
if i==0:
plt.ylabel('Actual')
plt.xlabel('Predicted')
plt.title(f'Model {i}')
plt.show();
print('Recall our Models:')
for n in notes:
print(f'\t{n}')
Recall our Models: Model 0: Three Convolution Layers with Dropout, Training Dataset: RGB Model 1: Four Convolution Layers with Dropout, Training Dataset: RGB Model 2: Three Convolution Layers with Dropout and first-layer batch normilization, Training Dataset: RGB Model 3: Three Convolution Layers with Dropout and first-layer batch normilization, Training Dataset: RGB (Augmented) Model 4: VGG16 Transfered at block3_pool Layer with two dense layers, Training Dataset: RGB Model 5: Autoencoder!!, Training Dataset: RGB
Note: There is some variablity between runs depending on the randoms seeds. This comparisons is based on several runs. Models 0, 1, and 3 all performed well. Model 5 consistently show the best performace.
"It is good for a professional to be reminded that his professionalism is only a husk, that the real person must remain an amateur, a lover of the work."
--May Sarton
Areas for development:
Data Preprocessing
Model
Methodology
Model
Data
Note:
There was some variablity between runs depending on the randoms seeds. This comparison is based on multple runs, for which Models 0, 1, 3 and 5 all performed well. Model 5 consistently shown the best performace.
Use libraries to visuallize model.
!pip install visualkeras &> /dev/null
#from keras.utils.vis_utils import plot_model
#plot_model(models[5], to_file=f'{path}/model_5_plot.png', show_shapes=True, show_layer_names=True)
#show our best model
import visualkeras
i=5
print('='*50)
print(f'Model {i}')
print('='*50)
visualkeras.layered_view(models[i], legend=True)
================================================== Model 5 ==================================================
# Compare with next best model
i=1
print('='*50)
print(f'Model {i}')
print('='*50)
visualkeras.layered_view(models[i], legend=True)
================================================== Model 1 ==================================================
Let's explore how our selected model might improve with Out-of-core learning.
"We are as Gods... So we might as well get good at it." --Stewart Brand
model = models[5]
history = model.fit(new_images,
new_labels,
batch_size = 32,
callbacks = callbacks,
validation_data=(my_validation_images, validation_labels),
epochs = 20,
verbose = 1)
history = history.history
Epoch 1/20 314/314 [==============================] - 14s 27ms/step - loss: 0.0685 - accuracy: 0.9789 - val_loss: 0.0675 - val_accuracy: 0.9778 Epoch 2/20 314/314 [==============================] - 10s 32ms/step - loss: 0.0673 - accuracy: 0.9773 - val_loss: 0.0711 - val_accuracy: 0.9737 Epoch 3/20 314/314 [==============================] - 10s 31ms/step - loss: 0.0673 - accuracy: 0.9783 - val_loss: 0.0709 - val_accuracy: 0.9760 Epoch 4/20 314/314 [==============================] - 8s 27ms/step - loss: 0.0667 - accuracy: 0.9764 - val_loss: 0.0662 - val_accuracy: 0.9754 Epoch 5/20 314/314 [==============================] - 8s 25ms/step - loss: 0.0654 - accuracy: 0.9775 - val_loss: 0.0675 - val_accuracy: 0.9758 Epoch 6/20 314/314 [==============================] - 6s 20ms/step - loss: 0.0638 - accuracy: 0.9781 - val_loss: 0.0645 - val_accuracy: 0.9772 Epoch 7/20 314/314 [==============================] - 8s 25ms/step - loss: 0.0632 - accuracy: 0.9779 - val_loss: 0.0640 - val_accuracy: 0.9774 Epoch 8/20 314/314 [==============================] - 6s 20ms/step - loss: 0.0646 - accuracy: 0.9772 - val_loss: 0.0647 - val_accuracy: 0.9760 Epoch 9/20 314/314 [==============================] - 7s 21ms/step - loss: 0.0629 - accuracy: 0.9790 - val_loss: 0.0661 - val_accuracy: 0.9754 Epoch 10/20 314/314 [==============================] - 6s 20ms/step - loss: 0.0630 - accuracy: 0.9781 - val_loss: 0.0618 - val_accuracy: 0.9776 Epoch 11/20 314/314 [==============================] - 7s 21ms/step - loss: 0.0606 - accuracy: 0.9799 - val_loss: 0.0695 - val_accuracy: 0.9747 Epoch 12/20 314/314 [==============================] - 6s 20ms/step - loss: 0.0600 - accuracy: 0.9799 - val_loss: 0.0673 - val_accuracy: 0.9770 Epoch 13/20 314/314 [==============================] - 7s 21ms/step - loss: 0.0622 - accuracy: 0.9784 - val_loss: 0.0648 - val_accuracy: 0.9760 Epoch 14/20 314/314 [==============================] - 6s 20ms/step - loss: 0.0600 - accuracy: 0.9773 - val_loss: 0.0743 - val_accuracy: 0.9733
plot_confusion_matrix(model, my_test_images)
(0.9842-0.9827)*100
82/82 [==============================] - 1s 8ms/step
precision recall f1-score support
0 0.9810 0.9908 0.9858 1300
1 0.9907 0.9808 0.9857 1300
accuracy 0.9858 2600
macro avg 0.9858 0.9858 0.9858 2600
weighted avg 0.9858 0.9858 0.9858 2600
0.14999999999999458
Training with augmented data shows that this model will performance can improve as additional training data is provided.
Background Resourses:
[0] Poostchi, Mahdieh et al. “Image analysis and machine learning for detecting malaria.” Translational Research 194 (2018): 36–55.
[1] Andrade, M.V., Noronha, K., Diniz, B.P.C. et al. The economic burden of malaria: a systematic review. Malar J 21, 283 (2022). https://doi.org/10.1186/s12936-022-04303-6
[2] https://en.wikipedia.org/wiki/Malaria
[3] Sarkar PK, Ahluwalia G, Vijayan VK, Talwar A. Critical Care Aspects of Malaria. Journal of Intensive Care Medicine. 2010;25(2):93-103. doi:10.1177/0885066609356052
[4] "5.1 Diagnosing Malaria (2015)". WHO Guidelines for Malaria. World Health Organization. 14 March 2023
[5] Delgado-Ortet M, Molina A, Alférez S, Rodellar J, Merino A. A Deep Learning Approach for Segmentation of Red Blood Cell Images and Malaria Detection. Entropy (Basel). 2020 Jun 13;22(6):657. doi: 10.3390/e22060657. PMID: 33286429; PMCID: PMC7517
[6] Brunton, S., & Kutz, J. (2019). Data-Driven Science and Engineering: Machine Learning, Dynamical Systems, and Control. Cambridge: Cambridge University Press. doi:10.1017/9781108380690
[7] Provenzi, E. Geometry of color perception. Part 1: structures and metrics of a homogeneous color space. J. Math. Neurosc. 10, 7 (2020). https://doi.org/10.1186/s13408-020-00084-x
Other Web Resourses:
https://docs.opencv.org/3.4/de/dbc/tutorial_py_fourier_transform.html
https://www.hindawi.com/journals/jece/2017/7948571/
https://wiki.seg.org/wiki/Spectral_decomposition
https://wires.onlinelibrary.wiley.com/doi/full/10.1002/widm.1474
https://www.ncbi.nlm.nih.gov/pmc/articles/PMC7517192/
https://en.wikipedia.org/wiki/Precision_and_recall
https://docs.celly.ai/supported-tests/malaria-thick-blood-film
https://docs.celly.ai/supported-tests/malaria-thin-blood-film
(see: Appendix 3: Core Summary)
Several models of varying layer complexity were compared. We found that model performance wasn’t necessarily a function of complexity. We also discovered that we could extract certain layers with image processing. While image processing could reduce training and model complexity, this also removed latent variables that might have been made available to the model. Image processing was kept to a minimum.
It was ascertained that certain specific image process was however valuable. Image rescaling helped to reduce input layer complexity. The addition of a small border improved data augmentation by removing artifacts due to zooming pixels on the edge of the image.
The proposed model maximizes accuracy yet balances complexity, memory usage, and execution time. The optimal model was discovered empirically by extracting layers and making informed assumptions[7] about the distributions of latent variables. A base CNN (Convolution Neural Network) model was adapted into the simplest possible AE (Auto-Encoder)[6] model then layers were augmented to maximize accuracy and precision. Model iterations were compared based on f1-scores.
The final model is best described as a DCNN (Deep Convolution Neural Network) however is referred to as ‘Model 5: Autoencoder!!’ in the code, due to the enormous 10-fold decrease in parameters and the initial success of the model. The model architecture consists of 4 main layers and 4 hidden cell layers.
The input layer utilizes batch-normalization to address batch variation and improve future out-of-core training. In this input layer we also perform a convolution over the entire image, replicating the results of 2D wavelet image experiments. The second main layer down-samples from the dense input using max pooling. The third main layer again down-samples and acts as the bottleneck feature of the model. The fourth main layer consists of a classification network and model output.
See: Model and Data Proposal->Model 5 Architecture
See also:
[6] Brunton, S., & Kutz, J. (2019). Data-Driven Science and Engineering
[7] Provenzi, E. Geometry of color perception. Part 1: structures and metrics of a homogeneous color space. J. Math. Neurosc. 10, 7 (2020). https://doi.org/10.1186/s13408-020-00084-x
The objective for this project was to build an accurate and efficient machine vision model that can improve the detection rate of a laboratory technician. A model was developed to identify whether or not a red blood cell in an image is infected with Malaria.
A CNN model was chosen based on the nature of the dataset--color images. Consideration was given to how additional training data could be used to build an additive model and how the addition of images might cause an imbalance in the loss function.
We minimized image processing to assure hidden features are available for hyper-tuning. Images are resized to reduce model complexity and a border added to improve data augmentation. Color Space transformations were evaluated and found to be isomorphic thus no further image processing was deemed necessary.
Reducing image processing helps to reduce overhead during file importing and improves out-of-core training. By normalization per additional training batch, we minimize side-effects due to changes in workflow and variations in lighting or evolving imaging technologies.
The model’s performance is exceptional and it features the ability to save and load the architecture/weights and will ease the deployment of the model. Additional software features such as web portal to upload images and retrieve results would need to be developed or adapted. Server daemons to manage, process and present images to the inference model will also be required. A portal interface for data curation and out-of-core training might also be considered.
This model does not stand-alone--it is an integrated part of a Clinical Decision Support System. Here we see the key stakeholders and show how they each interact with the machine learning model. The diagram above also shows where the data and decisions reside.
The key befits of this model and system architecture ought be evident by now. For the laboratory technician we improve their ability and speed to process samples. We provide accurate prediction to the physician to aid their treatment decision. We enable an expert collaborator to curate and encode data for out-of-core learning. Finally we provide model to machine learning operations that is easy to deploy, maintain, and grows with changing technologies.
We already detailed remaining challenges for deployment, namely, customer/expert portals and supporting server daemons. Other challenges such as containerization for cloud deployment remain. While out of scope for this project we’ve attempted to identify these areas as we transition from development to deployment.
#assertEnd()
logging.shutdown() # Only logs for complete runs will be saved.